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ABSTRACT 



This report presents findings from a national study of the 
relationship between different uses of educational technology and various 
educational outcomes. Data were drawn from the 1996 National Assessment of 
Educational Progress (NAEP) in mathematics, consisting of national samples of 
6,227 fourth graders and 7,146 eighth graders. Data include information on 
the frequency of computer use for mathematics in school, access to computers 
at home and in school, professional development of mathematics teachers in 
computer use, and the kinds of instructional uses of computers in the 
schools. The study finds that the greatest inequities in computer use are not 
in how often they are used, but in the ways in which they are used. Poor, 
urban, and rural students are less likely to be exposed to higher order uses 
of computers than nonpoor and suburban students. For both fourth and eighth 
grades, teachers of urban and rural students are less likely to have had 
professional development in technology than suburban teachers. There were few 
differences in the frequency of school computer use in either grade, although 
black fourth graders reported more frequent use than white fourth graders . 

Yet for both grades, black students were less likely to have a computer at 
school. In essence, the study found that technology could matter, but that 
this depended on how it was used. The size of the relationship between the 
various positive uses of technology and academic achievement was negligible 
for fourth graders, but substantial for eighth graders. Taken together, 
findings indicate that computers are neither a cure-all for problems facing 
the schools nor mere fads without impact on student learning. When used 
properly, computers may serve as important tools for improving student 
proficiency in mathematics and the overall learning environment of the 
school. An appendix discusses how the study was conducted. (Contains 2 
tables, 14 figures, and 23 references.) (SLD) 
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In this age of technologi- 
cal imperative, we do it 
simply because it can be 
done. Massive efforts are 
underway to convert 
traditional teaching to 
something that can be 
delivered via computer. 
Measuring success has 
been a simple matter: 
count the number of 
computers, divide that by 
the number of students, 
and report how the ratio 
of computers to students 
has advanced — and it is 
always advancing. Then 
close the report lament- 
ing that we don’t know 
much about the software 
being used on these 
computers, we don’t 
know how many are 
behind locked doors, we 
don’t know how many 
are broken, and we don’t 
know how many teach- 
ers really know how to 
use them (there are no 
assessments of teacher 
capability here). 

The Policy Informa- 
tion Center reported in 
May of 1997 what we did 
know at that time, in 
Computers and Class- 
rooms. We said then that 
the available data, and 
the research performed 
on it, did not tell us 
whether computer- 
delivered instruction 
actually improved perfor- 
mance. This report takes 
at least the first step in 
determining whether 
computer use is making 







a difference in math- 
ematics, and what kind 
of computer use has 
what kind of effect, 
on which groups 
of students. 

This study uses a 
national database, the 
1996 National Assess- 
ment of Educational 
Progress, and advanced 
analysis techniques, to 
isolate the effects of the 
computer from the 
myriad other factors 
involved in student 
achievement. The study 
was suggested to us 
by Education Week , 
and its author, Harold 
Wenglinsky, of the ETS 
Policy Information 
Center, has collaborated 
with the staff of Educa- 
tion Week throughout 
the analysis and writing 
phases of the report. 

In addition to telling 
us what he found, 
Harold Wenglinsky 
also tells us what 
further research must 
be conducted to learn 
more, and give the 
nation specific guidance 
in its efforts to raise 
educational achieve- 
ment through greater 
use of technology. 

Paul E. Barton 
Director 

Policy Information Center 



This project involved a 
close collaboration 
between myself and the 
many people working 
on Technology Counts , a 
yearbook on educational 
technology funded by 
the Milken Family 
Foundation and pub- 
lished by Education 
Week. Indeed, the study 
was inspired by a 
concern at Technology 
Counts that there was a 
lack of national informa- 
tion on the relationship 
between various aspects 
of educational technol- 
ogy and academic 
achievement. I am 
indebted to the Technol- 
ogy Counts staff for their 
ongoing and timely 
feedback on the project, 
particularly Craig Jerald, 
Virginia Edwards, and 
Jeffrey Archer. Such a 
collaboration between 
researchers and journal- 
ists, sometimes referred 
to as computer-assisted 
journalism, is a critical 
tool for making available 
to the public the fruits of 
statistical analyses of 
large-scale databases. 

Many others also 
contributed to this 
project. Stephen Gorman, 
John Barone, John 
Mazzeo, and Alfred 
Rogers facilitated access 
to the data. Henry Jay 
Becker, Ellen Mandinach, 
and Paul Barton pro- 
vided thorough and 
timely reviews of the 



draft report. Richard 
Coley assisted in pro- 
ducing the figures. Carla 
Cooper provided desk- 
top publishing services. 
Kirsty Brown did the 
editing, Kelly Gibson 
was the cover designer, 
and Jim Chewning 
coordinated production. 
Any errors of fact or 
interpretation included 
in this report, however, 
are my responsibility. 



SUMMARY OF FINDINGS 



This report presents 
findings from a national 
study of the relationship 
between different 
uses of educational 
technology and various 
educational outcomes. 
Data were drawn from 
the 1996 National 
Assessment of Educa- 
tional Progress (NAEP) 
in mathematics, con- 
sisting of national 
samples of 6,227 fourth- 
graders and 7,146 
eighth-graders. NAEP 
includes information 
on educational tech- 
nology, including: 

• frequency of com- 
puter use for math- 
ematics in the school; 

• access to computers 
and frequency of 
computer use in 
the home; 

• professional develop- 
ment of mathematics 
teachers in computer 
use; 

• kinds of instructional 
uses of computers by 
mathematics teachers 
and their students 

NAEP also includes 
various outcomes of 
education that are highly 
valued by policymakers 
and the public at large, 
including: academic 
achievement in math- 
ematics as measured 
by a standardized 
test; and the social 
environment of the 

O 




school, including student 
tardiness and absentee- 
ism, teacher absentee- 
ism, teacher morale, 
and student regard for 
school property. 

The study first 
compared the informa- 
tion about educational 
technology among 
different groups of 
students to discover any 
possible inequities in 
technology use. It found 
that the greatest inequi- 
ties did not lie in how 
often computers were 
used, but in how they 
were used. 

• For eighth-graders 
only, Black students 
were less likely to be 
exposed to higher- 
order uses of comput- 
ers and more likely to 
be exposed to lower- 
order uses than White 
students. Similarly, 
poor, urban, and rural 
students were less 
likely to be exposed 
to higher-order uses 
than non-poor and 
suburban students. 

• For both eighth- and 
fourth-graders, urban 
and rural students 
were less likely to 
have mathematics 
teachers who had 
received professional 
development in 
technology over the 
last five years than 
suburban students. 



For eighth-graders, 
poor students were 
less likely to have 
mathematics teachers 
who had received 
such professional 
development than 
non-poor students. 

• There was little 
difference in the 
frequency of school 
computer use in either 
grade, except that 
Black fourth-graders 
reported more fre- 
quent school com- 
puter use than White 
fourth-graders. 

• For both eighth- and 
fourth-graders, Black 
students were less 
likely to have access 
to a home computer 
than White students. 
Similarly, poor, urban, 
and rural students 
were less likely to 
have access to a 
home computer 
than non-poor and 
suburban students. 

The study then 
related these different 
aspects of technology 
to the educational 
outcomes, taking into 
account differences in 
student bodies and the 
other characteristics of 
the schools being ana- 
lyzed. In essence, the 
study found that technol- 
ogy could matter, but 
that this depended upon 



how it was used. For 
eighth-graders, the study 
found that: 

• Teacher’s professional 
development in 
technology and the 
use of computers to 
teach higher-order 
thinking skills were 
both positively related 
to academic achieve- 
ment in mathematics 
and the social envi- 
ronment of the school. 

• The frequency of 
home computer use 
was positively related 
to academic achieve- 
ment and the social 
environment of 

the school. 

• The use of computers 
to teach lower-order 
thinking skills was 
negatively related to 
academic achievement 
and the social envi- 
ronment of the school. 

• The frequency of 
school computer use 
was unrelated to the 
social environment 
of the school and 
negatively related to 
academic achievement. 

For fourth-graders, the 
study found that: 

• Using computers for 
learning games was 
positively related to 
academic achievement 
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and the social environ- 
ment of the school. 

o By increasing the 
tendency of teachers 
to use computers for 
learning games, profes- 
sional development 
of teachers was also 
positively related to 
academic achievement 
and the social environ- 
ment of the school. 

o The frequencies of 
home and school 
computer use were 
negatively related to 
academic achievement 
and the social environ- 
ment of the school. 

The size of the relation- 
ships between the various 
positive uses of technology 
and academic achievement 
was negligible for fourth- 
graders but substantial for 
eighth-graders. For fourth- 
graders, professional 
development and using 
computers for learning 
games each contributed 
about a tenth of a grade 
level of academic 
achievement, or the 
equivalent of a few 
weeks of instruction. For 
eighth-graders, however, 
professional development 
and using computers for 
higher-order thinking skills 
were each associated with 
more than a one-third 
of a grade level increase. 



These findings 
come with certain 
caveats. First, the data 
were collected at a 
single point in time; 
the aspects of technol- 
ogy studied here 
occurred at the same 
time as the educational 
outcomes of interest. 
Thus, it may be that 
high-achieving stu- 
dents are more likely 
to use technology in 
certain ways rather 
than that these uses of 
technology promote 
high levels of aca- 
demic achievement. 
Second, while the 
study takes into 
account some charac- 
teristics of teachers, 
it does not take into 
account their overall 
tendency to teach in 
certain ways, such as 
to teach higher-order 
thinking skills. It may 
be that computers are 
but one medium 
among many that 
teachers use to teach 
higher-order thinking 
skills, and that all 
of these media are 
conducive to high 
levels of academic 
achievement. 

All of this suggests 
that computers are 
neither cure-alls for 
the problems facing 
schools, nor mere fads 
that have no impact 



on student learning. 
Rather, when they are 
properly used, comput- 
ers may serve as impor- 
tant tools for improving 
student proficiency 
in mathematics, as 
well as the overall 
learning environment 
in the school. 



INTRODUCTION 



In recent years, educa- 
tional technology has 
become a cornerstone 
for state and federal 
efforts to improve the 
performance of the 
nation’s school children. 
“Educational technology” 
generally refers to the 
introduction of comput- 
ers and related pieces 
of equipment to the 
classroom. Across the 
nation, governors have 
targeted a large share of 
their education budgets 
to making computers 
more available in 
schools. George Pataki, 
governor of New York, 
and Pete Wilson, gover- 
nor of California, have 
asked their state legisla- 
tures for millions of 
dollars to purchase new 
computers for schools 
and train teachers to use 
them. Some states are 
taking a top-down 
approach, bringing the 
decision-making skills 
and computer expertise 
of the state government 
to bear to move technol- 
ogy to the classroom as 
quickly and inexpen- 
sively as possible. West 
Virginia, for example, 
purchased computers in 
bulk and sent them to 
school districts; the state 
is also responsible for 
teacher training. Other 
states are providing the 
funds but allowing 
school districts to make 
the decisions. Kentucky, 
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for instance, required 
each school district to 
develop its own tech- 
nology plan and make 
its own purchases of 
software and hardware 
with state funds (see 
White, 1997). 

State efforts to 
increase access to 
technology have been 
complemented by 
federal action. In 1996, 
President Clinton and 
Vice President Gore 
announced the Tech- 
nology Literacy Chal- 
lenge. The overarching 
goal of the program 
was to ensure that all 
students had access to 
educational technology. 
Clinton and Gore 
identified four pillars 
of technological lit- 
eracy, namely modern 
computer hardware, 
the Internet, teacher 
development, and high- 
quality software and 
online resources. The 
Technology Literacy 
Challenge Grants have 
been awarded to states 
to help them provide 
these elements. This 
and other funding from 
the Federal Govern- 
ment has resulted in a 
federal contribution of 
25 percent of all dollars 
spent on educational 
technology. To further 
assist in meeting the 
goal of connecting 
students to the 
Internet, the Clinton 



Administration initiated 
the E-Rate program, 
which requires the 
telecommunications 
industry to contribute 
dollars to a fund to 
support school and 
library Internet access 
(U.S. Department of 
Education, 1996). 

Yet with all of this 
money being spent on 
educational technology, 
many policymakers are 
wondering about the 
evidence on its effective- 
ness. The purpose of 
providing technology 
to schools is to improve 
student academic perfor- 
mance and other educa- 
tional outcomes, not to 
provide state-of-the-art 
equipment for its own 
sake. What types of 
investment in technol- 
ogy most improve 
academic achievement? 
Does technology affect 
important educational 
outcomes at all? Unfortu- 
nately, for all of the 
investment in educa- 
tional technology, there 
is a surprising lack of 
hard data on its effects. 
State data collection 
efforts are sketchy at 
best, and the federal 
government does not 
collect national data 
expressly for the 
purpose of evaluating 
educational technology. 
Prior research on com- 
puters in the classroom 
consists primarily of 



evaluations of exem- 
plary programs; these 
studies have significant 
methodological prob- 
lems, and it is highly 
questionable whether 
the programs can 
be replicated on a 
large scale. 

Fortunately, a great 
deal of information can 
be gleaned from the 
most recent National 
Assessment of Educa- 
tional Progress (NAEP) 
for which data are now 
available. NAEP consists 
of tests in various 
subject areas adminis- 
tered to national 
samples of students. It 
is generally given every 
two years, and is used 
to measure trends in 
student performance 
over time, as well as 
between subgroups of 
students such as males 
and females. NAEP also 
includes questionnaires 
given to students, their 
teachers, and their 
principals, to put the 
NAEP test scores in their 
educational context. The 
1996 NAEP, for the first 
time, included detailed 
questions on technology 
in its questionnaires. By 
examining these ques- 
tions and relating them 
to students’ test scores 
and other characteristics 
of students, teachers, 
and schools, it is 
possible to learn a 
great deal about the 
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effectiveness of technol- 
ogy across the nation. 

This report presents 
the results of a national 
study of technological 
effectiveness using the 
NAEP in mathematics. It 
finds that technology can 
indeed be effective in 
increasing mathematics 
achievement and other 
educational outcomes, 
but that this effectiveness 
depends upon how 
technology is used. 

The report is organized 
as follows: 

o Chapter One presents 
the debate on techno- 
logy’s effectiveness. It 
then discusses what 
the prior research says 
on this issue, and finds 
it to suffer from meth- 
odological problems 
that limit its usefulness 
to the debate. 

o Chapter Two presents 
descriptive information 
about access to and 
use of educational 
technology by our 
nation’s schoolchil- 
dren, particularly for 
mathematical tasks. 

It finds that large 
government invest- 
ments in technology 
have eliminated gaps 
in access to and 
frequency of use of 
computers, but that 
large gaps in teacher 
preparation and 
the ways in which 



technology is used 
in the classroom 
remain. 

o Chapter Three 
presents the results 
of a statistical analy- 
sis of the relationship 
between technology, 
academic achieve- 
ment in mathematics, 
and other educational 
outcomes. It finds 
that the usefulness of 
technology depends 
upon how the tech- 
nology is used; some 
uses are positively 
related to academic 
achievement and 
other educational 
outcomes, while 
others are not. 

o Chapter Four dis- 
cusses the method- 
ological shortcom- 
ings of this study 
and suggests some 
implications of the 
study’s findings for 
policymakers and 
practitioners. 




Chapter One: 
The Debate on 
Technology’s 
Effectiveness 



The penetration of 
computers into the life 
of schools is a fact few 
could deny. The avail- 
ability of computers 
nationally has increased 
dramatically in recent 
years, from a computer 
for every 125 students 
in 1983 to a computer 
for every 9 students in 
1995 (Glennan & 
Melmed, 1996). Many 
schools now have one 
computer for every 
two students. Yet the 
response of educators 
and practitioners has 
been mixed. Some have 
been quite enthusiastic 
about technology, 
seeing it as the tool 
most needed to facilitate 
more comprehensive 
educational reform. 
Others see it as a 
passing fad, more a 
distraction from school 
reform than anything 
else. Not all advocates 
and critics subscribe to 
the views paraphrased 
below; these are simply 
illustrations of a few of 
the prevailing views 
regarding technology. 

The Case for Technology 

Advocates of tech- 
nology argue for using 
technology for the 
wholesale transformation 
of the classroom. 1 They 



note that computers and 
supporting technologies 
have any number of 
uses. These are generally 
of five types: support for 
individual learning, 
group learning, and 
instructional manage- 
ment; communication; 
and administration. 
Applications to indi- 
vidual learning include 
drilling students on 
particular skills, using 
CD-ROMs or the Internet 
to find resources not 
available in the school, 
communicating with 
experts, word-process- 
ing, providing assistance 
in computations, and 
demonstrating simula- 
tions of mathematical 
or scientific concepts. 
Group learning applica- 
tions include using 
e-mail to support group 
communication, using 
presentation software to 
allow group presenta- 
tions on a project, and 
providing collaboration 
in collecting and analyz- 
ing data. Applications to 
instructional manage- 
ment include integrating 
standards and assess- 
ments, managing student 
portfolios, and develop- 
ing individual student 
learning plans. Commu- 
nications applications 
include communicating 
to remote locations such 



as rural schools and 
improving communica- 
tion among students, 
teachers, and parents. 
Finally, applications to 
administrative functions 
include supporting 
attendance and account- 
ability functions. 2 
Clearly, using technol- 
ogy in all of these ways 
would result in the 
penetration of technol- 
ogy into every aspect 
of the school day, 
theoretically making 
conventional teaching 
techniques obsolete. 

This vision has 
been applied, to some 
degree, in various 
innovative schools. 
Blackstock Junior High 
School in California, for 
instance, developed 
“smart” classrooms, in 
which each student has 
a computer, and these 
are on a network. The 
teacher can thus allow 
students to work either 
independently or in 
groups over the net- 
work. The school also 
developed the Tech Lab 
2000 to provide even 
more sophisticated 
facilities, such as Com- 
puter Assisted Design 
software, a Computer 
Numerically Controlled 
flexible manufacturing 
system, and even a 
satellite dish. Even 



!The case of technology presented here draws heavily on Glennan & Melmed (1996). Also, 
see p^ge 9 for examples of studies supporting technology. 

2 An alternate schema for categorizing uses of technology can be found in Becker (1994a). 



non-smart classrooms 
are technology-rich, 
with 10 computers and 
two printers per room. 
Other innovative 
schools have similarly 
technology-rich layouts. 
Northbrook Middle 
School in Texas has 
one computer for every 
two students; each 
classroom has five to 
six computers, com- 
plete with CD-ROM 
capabilities, a school- 
wide network, and 
Internet access. 

The primary indica- 
tor used by technology 
advocates to assess 
whether a school is 
using technology well 
is how much hardware 
and software has 
actually reached the 
classroom. Innovative 
schools are identified 
based upon their 
student-to-computer 
ratios, the sophistica- 
tion of their networks 
and peripherals, as 
well as the amount of 
money spent per-pupil 
on technology. In this 
view, transformation of 
educational practice 
requires a high level of 
access to technology; 
without such access, 
technology is educa- 
tionally marginalized, 
rather than being an 



essential feature of 
instruction. Thus a given 
school is not likely to be 
effective in its uses of 
technology unless there is 
a substantial technologi- 
cal infrastructure. 

In terms of the actual 
effects of technology on 
academic achievement 
and other educational 
outcomes, advocates 
assert that most uses of 
technology are beneficial, 
and that they can lead to 
improvements in most or 
all educational outcomes. 
They cite three pieces of 
evidence. First, they note 
that computers have been 
used to provide students 
with opportunities for 
drill and practice since 
the 1960s. This use, 
known as Computer 
Assisted Instruction (CAI) 
has been evaluated 
repeatedly, and the 
evaluations suggest that 
CAI students demonstrate 
higher levels of academic 
achievement than their 
counterparts who do not 
have CAI. Second, advo- 
cates note that more 
recent uses of technol- 
ogy, such as those that 
support higher-order 
thinking skills, have not 
been as widely evaluated, 
but the few evaluations 
done, such as that of 
Apple Computers of 



Tomorrow, show 
promising results. 

Finally, they note that 
technology-rich schools 
seem to evince gains in 
achievement and other 
educational outcomes 
such as student motiva- 
tion and teacher morale. 

The Case against 
Technology 

Others have been 
quite critical of technol- 
ogy. They agree with the 
advocates that the goal 
of spreading educational 
technology is to use it 
to transform instruction; 
the critics, however, 
view this effort as being 
doomed to failure. 3 Their 
indicator for the success 
of technology programs 
is not the amount of 
technology available, but 
whether teachers are 
willing to use it. No 
matter how many 
computers are available 
in the classroom, if 
teachers are unwilling to 
use them for instruction, 
they are unlikely to have 
much impact on stu- 
dents. And, in the view 
of the critics, teachers 
are by-and-large not 
making use of comput- 
ers. Schools tend to have 
a few teachers and 
administrators who are 



advocates of technology 
and use it widely. The 
majority of teachers, 
however, use technol- 
ogy as little as possible; 
they only do as much 
with computers as 
administrators require 
of them. 

The critics do not 
claim that teacher 
obstruction of technol- 
ogy is detrimental to 
schooling, but quite the 
opposite. They regard 
technology as unlikely 
to improve academic 
achievement or indeed 
any other educational 
outcome of students. In 
support of this view 
they present three 
pieces of evidence: the 
history of technological 
innovation, research on 
the cost-effectiveness 
of CAI, and evidence 
on theories of learning. 
First, they note that 
teachers have histori- 
cally been resistant to 
technological innova- 
tions when those inno- 
vations have made it 
more difficult for them 
to get through the 
typical school day. Early 
technologies, such as 
blackboards and desks, 
were supported by 
teachers because they 
made it easier for 
them to manage the 



3 The case against technology presented here draws heavily on Cuban (1986) and Cuban (1993). See pages 9 and 10 of this 
report for examples of studies criticizing technology. 
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classroom and convey 
information. Later 
innovations, such as 
films and educational 
television were resisted 
because they would 
undermine the teacher- 
student relationship; to 
the extent that these 
technologies would 
substitute for teachers, 
teachers would have 
less time to interact 
with students. Indeed, 
the only situations 
in which teachers 
embraced this technol- 
ogy were those in 
which teachers wanted 
to reduce interaction 
with students; for 
instance, when teachers 
wanted to waste time 
during low-energy 
periods of the day 
(e.g. after lunch) they 
might show a film. 
Computers, the critics 
claim, pose these same 
issues and therefore 
will be marginalized 
or not used at all 
by teachers. 

Second, research 
on the cost-effective- 
ness of CAI shows that 
while there may be 
some gains to academic 
achievement, these are 
not proportionate to 
the costs of buying and 



maintaining computers; 
tutoring, for instance, was 
found to produce greater 
gains for less money. 

Finally, cognitive 
theories of education 
suggest that learning 
may include a social, 
noncognitive element. 
Students learn not only 
because they process 
information, but also 
because of the complex 
reinforcements they 
receive from teachers 
and the socialization 
process in which learning 
is embedded. As comput- 
ers move from being 
mere supplements to 
being the core of the 
learning environment, 
they limit opportunities 
for social interaction, thus 
interfering with the 
learning process. Hence 
computers should not be 
expected to increase 
student learning and, in 
fact, may decrease it. 

The Evidence on 
Technology’s Effectiveness 

There are some 
problems with the asser- 
tions made by both 
advocates and critics 
regarding the relationship 
between technology 
and educational 



outcomes. With regard 
to the advocates, the CAI 
evaluations of drill 
and practice computer 
applications, the evalua- 
tion of higher-order 
applications and the 
data from the exemplary 
programs suffer from 
certain methodological 
problems that raise 
questions about their 
findings . 4 First, they treat 
technology as an undif- 
ferentiated characteristic 
of schools and class- 
rooms. No distinction is 
made between different 
types of technology 
program, such as pro- 
grams that involve 
intensive professional 
development or those 
that focus on higher- 
order thinking skills. 
Without differentiating 
between program 
components, it is not 
clear which are 
responsible for overall 
program effectiveness; it 
may be that some uses 
of technology are more 
productive than others. 
Second, these evalua- 
tions focus on a school 
or school district, but 
never on statewide or 
nationwide uses of 
technology. It may be 
that what works well in 



Chicago will not work 
well in New York City. 
Third, evaluations often 
use a poor measure of 
academic achievement; 
many use tests that have 
been developed for that 
particular study and 
have not been validated. 
Fourth, the students in 
technology-rich environ- 
ments may be very 
different from those in 
more conventional 
settings. Many of these 
studies do not randomly 
assign students to 
different environments, 
and some do not even 
have a group of stu- 
dents in a conventional 
environment to whom 
to compare the students 
using technology. 
Apparent gains in 
achievement may there- 
fore be attributable to 
the student rather than 
the school. These prob- 
lems make the conclu- 
sion from these studies, 
that technology is 
positively associated 
with academic achieve- 
ment, questionable . 5 

The evidence of the 
critics also raises diffi- 
culties. The historical 
argument that teachers 
obstruct technology 
because it is not useful 



4 For reviews of CAI, see Kulik & Kulik (1991), Niemiec et al. (1987), Liao (1992), and Christmann, Badgett & Lucking (1997). 
The reviews generally find significant differences in academic achievement and other outcomes between students exposed to CAI 
and those exposed to conventional teaching techniques, but find that these differences vary dramatically depending upon study 
methodology, with some studies finding no difference or even a negative relationship. 

5 For an example of a critique of CAI research, see Colorado (1988). 
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assumes that computers 
have as little use for 
education as film strips 
or educational television. 
To the extent that com- 
puters differ from these 
technologies, they 
should be embraced 
by teachers. Indeed, 
research on teacher 
obstruction indicates that 
the inability of many 
teachers to use comput- 
ers is based upon a lack 
of resources supporting 
teacher use of technol- 
ogy, not a Luddite 
feeling among teachers 
that technology is 
useless. Henry J. Becker 
(1994b) analyzed data 
from one of the most 
comprehensive surveys 
of teacher technology 
use, the International 
Association for the 
Evaluation of Educa- 
tional Achievement 
survey of 1989. Among 
the more than 1,000 
teachers surveyed, 

Becker did find both low 
levels of computer use 
and low levels of effec- 
tive use; only one of six 
teachers used computers 
regularly in the class- 
room, and of these, a 
majority used them 
mostly for drill and 
practice, avoiding more 
complicated applica- 
tions. (Just 16 percent of 



mathematics teachers 
used computers for 
conveying “understand- 
ing of numerical rela- 
tionships” as opposed to 
58 percent using them 
for “student mastery 
of computation.”) 

Yet when he tried to 
account for different 
types and levels of 
computer use by teach- 
ers, Becker found that 
teachers using comput- 
ers more effectively 
were more likely to 
work in schools offering 
high levels of teacher 
development on com- 
puters and having 
technology coordinators 
available to assist 
teachers with ongoing 
problems. A more 
recent study found that 
the primary reason 
teachers were afraid of 
technology was a lack 
of experience using 
computers (Rosen & 
Weil, 1995). Thus 
teacher obstruction is 
not an insurmountable 
barrier to computer 
use; teachers simply 
need to be trained and 
supported before they 
will feel ready to 
use them. 

The critics also 
argue that the CAI 
studies show computers 
to be less cost-effective 



than tutoring, and that 
technology-based educa- 
tion undermines the 
social, noncognitive 
components of learning. 6 
The problems with CAI 
studies have already 
been discussed; regard- 
ing the social aspects of 
learning, it can only be 
said that the impact of 
computers on learning 
depends upon how they 
are used. If they substi- 
tute entirely for human 
interaction, then they 
may indeed undermine 
social components of 
instruction. On the 
other hand, if they 
are one important tool 
among many, they 
will still permit a great 
deal of face-to-face 
contact between stu- 
dents and teachers. 

At this point, then, 
the debate on tech- 
nology’s effectiveness 
seems unresolved, 
leaving policymakers 
wondering about how 
to use technology or, 
indeed, whether to invest 
in it at all. Research in 
support of technology 
seems to consist of small- 
scale studies with serious 
methodological problems; 
research against technol- 
ogy is based upon 
unproved assertions 
that computers are 



no different from 
film strips. 

To some degree, 
the paucity of clear evi- 
dence on technology’s 
effectiveness stems from 
difficulties inherent in its 
study. As Mandinach 
and Cline (in press) 
have noted, the lack of 
controlled experiments 
in this area reflects the 
fact that technology is 
not introduced over 
a well-defined period 
of time. Most experi- 
ments involve a specific 
intervention with a clear 
beginning and end. The 
introduction of technol- 
ogy, however, involves 
obtaining hardware and 
software, training teach- 
ers, modifying curricula, 
and maintaining the 
technology, among other 
things. This process is 
therefore an ongoing 
one, without a clear 
starting or stopping 
point. Mandinach and 
Cline suggest that evi- 
dence on technology’s 
effectiveness will have 
to be drawn from various 
nonexperimental 
approaches, using 
multiple methods at 
various levels (e.g. the 
classroom, the school, 
the district). 



6 The study showing the differential cost-effectiveness of various interventions is reported in Levin, Desther & Meister (1986). 
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This report repre- 
sents one method of 
studying technology 
that can, perhaps, help 
break the statemate 
between technology’s 
advocates and critics. 

It presumes that the 
reality of technology’s 
effectiveness lies some- 
where between the 
optimism of advocates 
and the pessimism of 
critics. Some uses of 
technology are prob- 
ably conducive to 
academic achievement 
and other positive 
educational outcomes, 
while other uses of 
technology are not. 

The report analyzes 
national data on tech- 
nology that can distin- 
guish between practices 
that are effective and 
those that are not, while 
also addressing some 
of the shortcomings of 
prior research. Before 
relating patterns of 
technology use to 
academic achievement, 
however, it will be 
useful to examine 
those patterns. 
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Chapter Twos 
Patterns of 
Educational 
Technology 
Use 




The purpose of this 
chapter is to describe 
how computers are 
used by students and to 
compare usage between 
different types of stu- 
dents. Four key indica- 
tors of computer use are 
discussed here: 

o student access to 
computers in school 
for mathematical 
tasks; 

o student access to 
computers and 
frequency of com- 
puter use at home; 

o preparedness of 
mathematics teachers 
in computer use; 

o the ways in which the 
mathematics teachers 
and their students use 
computers. 

In addition to knowing 
the state of the nation in 
regard to these four 
indicators, it is impor- 
tant to know how they 
are distributed among 
different types of stu- 
dents. There may be 
large differences in 
computer access among 
students, and these 
differences may have 
important policy impli- 
cations; if minority 
students, for instance, 
have less access to 
teachers who have 
received training on 



computers, it might be 
important to increase 
teacher training in 
schools with high 
proportions of minority 
students. Six types of 
subgroups of students 
are particularly worth 
examining, as they 
appear frequently in 
policy discussions: 

O Ethnicity 
o Gender 

° Region of the country 
o Governance of school 
(public vs. private) 
o Economic status (poor 
vs. non-poor) 
o Community status 
(urban, suburban, 
rural) 

Data and Measures 

The state of technol- 
ogy use can be mea- 
sured from data from the 
National Assessment of 
Educational Progress of 
1996 (NAEP). NAEP has 
been administered 
regularly since 1969 to 
nationally representative 
samples of students of 
various ages and grade 
levels. The core of NAEP 
is an assessment that has 
been given in various 
subject areas including 
mathematics, science, 
reading, geography, and 
writing. NAEP also 
includes background 
questionnaires com- 
pleted by students, the 
principals in their 
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schools, and the teach- 
ers in the relevant 
subject area. Students 
and teachers are asked 
about their social 
background, their 
experiences in school 
generally, and their 
experiences in the 
particular subject area; 
principals are asked 
primarily about school 
policies and practices. 
While earlier administra- 
tions of NAEP did 
include a few questions 
about technology, only 
the 1996 administrations 
included sufficient 
information to measure 
the four indicators listed 
above. In 1996 NAEP 
was administered to 
fourth-, eighth- and 
twelfth-graders in 
mathematics and 
science. This report 
describes technology 
uses among the 6,627 
fourth-graders and 7,146 
eighth-graders who took 
the core NAEP assess- 
ment in mathematics. 

Student access to 
computers in school is 
measured by frequency 
of computer use. Often 
researchers use the ratio 
of computers to students 
or the level of spending 
as an indicator of access. 
Yet computers may be 
sitting in closets, or in 
classrooms unused. 

Thus, to avoid questions 
of whether school 
ownership of computers 



is the same as student 
access, frequency of use 
by students is measured 
instead; use weekly or 
more often is deemed to 
be true access because 
less frequent use sug- 
gests that computers act 
more as filler than as 
part of the curriculum. 

For this study, student 
reports of computer use 
rather than teacher 
reports are used, because 
of how the relevant 
questions are asked in 
NAEP. Students are 
asked: “When you do 
mathematics at school 
how often do you... use a 
computer?” Teachers are 
asked “How often do 
students in this class... use 
a computer?” Students 
are thus responding with 
the frequency with which 
they use computers 
specifically for mathemat- 
ics. Teachers are respond- 
ing with the frequency 
with which students in 
their class are using 
computers. For eighth- 
graders, the teacher most 
likely teaches mathemat- 
ics only, and so the 
responses would reflect 
student computer use in 
mathematics only, but for 
fourth-graders the 
teacher probably teaches 
the same students in a 
variety of subjects, and 



so the responses would 
reflect computer use 
more generally, thus 
overstating mathemati- 
cally oriented use. 
Student responses are 
measured to avoid this 
difficulty. Student 
access to a computer at 
home is also measured 
by frequency of use. 
Whether students’ 
families own computers 
is also included, 
because the NAEP 
question on frequency 
of home use includes 
the family not owning 
a computer as a choice. 
This information comes 
from the student 
questionnaire . 7 

Teacher prepared- 
ness is measured by 
whether teachers have 
received professional 
development in com- 
puter use. Presumably, 
teachers who receive 
training on computers 
are better prepared than 
those who do not; the 
Becker study reinforces 
this notion, finding that 
teachers who receive 
training that is subject- 
specific or tool-oriented 
are indeed exemplary 
users of educational 
technology. The alterna- 
tive is to use teachers’ 
perceptions of their 
preparedness, which is 



asked in NAEP. Yet these 
perceptions of prepared- 
ness might not reflect 
reality; teachers could 
think they are prepared 
but may not be. Also, 
changing such percep- 
tions is not as easy as 
funding teacher training; 
the perception measure 
thus has less policy 
relevance. The teacher 
development informa- 
tion comes from 
teacher reports. 

Finally, the ways in 
which computers are 
used is measured from a 
NAEP question that 
suggests four possible 
uses. These are: “drill 
and practice,” “demon- 
stration of new topics in 
mathematics,” “playing 
mathematical/learning 
games,” and “simulations 
and applications.” To 
compare subgroups, the 
responses that indicate a 
higher-order thinking 
activity are placed in 
one category and those 
that indicate a lower- 
order thinking activity 
are placed in another. 
For fourth-graders, drill 
and practice is deemed 
the lower-order activity 
and learning games the 
higher-order activity. 
While many learning 
games may simply be 
a form of drill and 



7 Some research raises questions about the validity of even student reports (Becker, 1994c). 
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practice and therefore 
lower-order, other 
learning games may 
be higher-order. It is 
unlikely that there will 
be much opportunity 
for the demonstration 
of new topics or 
simulations and appli- 
cations because so 
much of fourth-grade 
mathematics is compu- 
tational. Thus, only the 
learning games cat- 
egory is likely to 
include substantial 
application of higher- 
order skills. 

For eighth-graders, 
drill and practice are 
again deemed lower- 
order, but simulations 
and applications are 
deemed higher-order. 
Because eighth-grade 
mathematics does 
introduce algebraic 
and geometric con- 
cepts (among others), 
there is likely to be 
ample opportunity to 
apply these concepts. 
Learning games 
probably function as 
busywork in the 
eighth-grade context, 
and so are neither 
higher-order nor 
lower-order; new 
topics are probably 
higher-order or lower- 
order depending 
upon the topic, and 
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so are also not 
classified either way 
(See Appendix for 
additional information 
on how this part of the 
study was conducted.) 8 

Access to School 
Computers 

Frequency of com- 
puter use in school 
among fourth-graders is 
presented in Figure 1. 
Overall, 33.2 percent of 
students reported using 
computers in school for 
mathematics work once 
a week or more often. 
Access was somewhat 
greater for Black stu- 
dents than for other 
ethnic groups; fully 41.9 
percent of Black stu- 
dents reported frequent 
use (as opposed to 32.3 
percent of Hispanic 
students and 31.7 
percent of White stu- 
dents). There was also 
some regional differen- 
tiation; students in the 
Southeast used comput- 
ers in schools more 
frequently than students 
in the Midwest. The 
other regions were 
somewhere in the 
middle. Finally, public 
school students used 
computers more fre- 
quently than students 



Figure 1: Percentage of Fourth-Graders Who [Report Using 
Computers at School at Least Once a Week 



u.s. 



■# 33 



Black 

Asian 

Hispanic 

White 



# 42 



— • 33 
32 



32 



Male 

Female 

School Lunch Eligible 
School Lunch Ineligible 







w o*+ 

A oo 




I 9 oo 




1 - 




w 

a oo 


|W ** 



Southeast 

Northeast 

West 

Central 






37 

• 35 

— • 34 
28 



Public 

Private 



F 73 # 34 
-• 28 



Urban 

Rural 

Suburban 



— • 35 
-• 33 

■# 32 



0 10 20 30 40 

Percentage of Students 



n 

50 



Note: Only values on opposite sides of the shaded area are statistically significantly 
different from one another. 



Source: 1996 NAEP Mathematics Assessment 



8 The distinction between higher-order and lower-order thinking is of long standing. For the case that technology’s virtues are 
revealed primarily in higher-order thinking tasks, see Means (1994). - 

o If) 



ERIC 




Figure 2: Percentage of Eighth-Graders Who Report Using 


Computers at School at Least Once a Week 
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attending private school. 
There were no differ- 
ences based upon 
gender, economic status, 
or community status. 

Eighth-graders 
appear to use computers 
about as frequently as 
fourth graders; 28.3 
percent reported using 
computers once a week 
or more often (Figure 
2). Unlike fourth- 
graders, however, there 
were no regional differ- 
ences. Males had higher 
levels of computer use 
than females. Poor 
students had higher 
levels of computer use 
than non-poor students. 
And students attending 
urban schools had 
higher levels of com- 
puter use than students 
attending rural schools. 
Also, among eighth- 
graders, private school 
students had higher 
levels of computer use 
than public school 
students. 

These differences 
suggest that traditionally 
disadvantaged groups 
do not lag in access to 
school computers. 
Compared to White, 
non-poor, and suburban 
students, those who are 
minority, poor, and 
urban find at least as 
much opportunity to use 
computers in school. 
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Access to Home 
Computers 

Overall, a majority 
of fourth-graders have 
access to a computer at 
home (58 percent, see 
Figure 3). This access 
varies greatly between 
groups. Ethnic differ- 
ences were quite large. 
Among White and Asian 
students, two-thirds had 
access to computers in 
the home as opposed to 
less than half of Black 
and Hispanic students. 
There were substantial 
regional differences. 

Less than half of the 
students attending 
schools in the Southeast 
had access to a home 
computer, as opposed to 
two-thirds of students in 
the Northeast; the other 
regions were in the 
middle. There were also 
differences based upon 
school governance, 
economic status, and 
community status; 
private school students 
were more likely to 
have home computer 
access (74 percent as 
opposed to 56 percent 
for public school stu- 
dents); non-poor stu- 
dents were more likely 
to have access (65.3 
percent as opposed to 
40.6 percent for poor 
students); and students 
attending suburban 



Figure 3: Percentage of Fourth-Graders with Access to Home 
Computers 
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Figure 4: Percentage of Eighth-Graders with Access to Home 


Computers 




u . s . — 


#64 


White 


-#69 


Asian 


— -#68 


Black 


#51 


Hispanic 


•$0 


Male 


-#66 


Female 


#63 


School Lunch Ineligible 


#72 


School Lunch Eligible 


#42 


Northeast 


— #68 


West 


#66 


Central 


#64 


Southeast 


#60 


Private 


#77 


Public 


«63 


Suburban 


— #69 


Urban 


#61 


Rural 


#6° 


1 

0 


1 r i i i 

20 40 60 80 100 




Percentage of Students 


Note: Only values on opposite sides of the shaded area are statistically significantly 


different from one another. 




Source: 1996 NAEP Mathematics Assessment 



schools were more 
likely to have access 
(63 percent as opposed 
to 52.7 percent and 51.7 
percent for urban and 
rural schools respec- 
tively). There were no 
gender differences. 

Eighth-graders 
showed essentially the 
same pattern (Figure 4). 
Overall, 64.3 percent 
reported a computer in 
the home. Black and 
Hispanic students were 
much less likely to have 
access than White and 
Asian students. Students 
from the Northeast had 
higher levels of access 
than students from the 
Southeast. Private school 
students had higher 
levels of access than 
public school students. 
Non-poor students had 
higher levels of access 
than poor students. And 
suburban students had 
higher levels of access 
than urban or rural 
students. In the eighth- 
grade case, however, 
males had higher levels 
of access than females. 

A separate question 
from how many stu- 
dents are in families that 
own computers is how 
many students use 
home computers regu- 
larly if they have them. 
For fourth-graders, 
among those students 
who have access to a 
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home computer, 32.5 
percent used them 
weekly or more often, 
about the same as the 
level of school use 
(Figure 5). Differences 
in frequency of use at 
home also mirrored 
those of frequency of 
use at school. Blacks 
showed higher levels of 
use than Whites and 
Hispanics; 53 percent of 
Blacks reported using 
computers once a week 
or more. There was 
some regional differen- 
tiation, with Southeast- 
ern and Western students 
using computers at home 
more than Midwestern 
students. Among students 
having computers in the 
home, public school 
students and poor stu- 
dents also used them 
more frequently than 
private school or non- 
poor students. There were 
no significant differences 
based upon gender or 
community status. 

Among eighth- 
graders with computers 
at home, the patterns 
were less interesting 
(Figure 6). In all, 46.6 
percent of eighth-graders 
reported using computers 
at home weekly or more 
often; this is a higher 
level than for fourth- 
grade home use or 
eighth-grade school use. 
There were, however, no 
significant differences 
between subgroups. 
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Figure 5: Percentage of Fourth-Graders Using the Computer at 
Home Once a Week or More 
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Figure 6: Percentage of Eighth-Graders Using the Computer at 
Home Once a Week or More 
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These patterns of 
home use suggest that 
inequities exist in the 
availability of computers 
at home, but not in their 
use. Poor, minority, and 
urban students are less 
likely to live in homes 
that possess computers 
in both fourth and 
eighth grade. However, 
for those families that 
own computers, student 
use tends to be as high 
or higher among disad- 
vantaged groups. 

Teacher Preparation 

Among fourth- 
graders, four out of five 
had teachers who 
reported receiving 
professional develop- 
ment in the use of 
computers in the previ- 
ous five years (Figure 
7). These percentages 
varied by region of the 
country, school gover- 
nance, student eco- 
nomic status and com- 
munity status. More 
public school teachers 
than private school 
teachers reported 
professional develop- 
ment, as did more 
teachers of non-poor 
students than teachers 
of poor students, and 
more suburban 
teachers than urban 
or rural teachers. 
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Of the regions of the 
country, the Southeast 
had the most profes- 
sional development 
(85.6 percent of students’ 
teachers) followed by 
the West and the Mid- 
west, with the Northeast 
trailing behind (with 
77.0 percent). There 
were no ethnic or 
gender differences. 

For eighth grade 
(Figure 8), the percent- 
age of students with 
teachers reporting 
professional develop- 
ment was somewhat 
lower, but still quite 
high, at 76 percent. 

Again the Northeast 
lagged behind the other 
regions of the country, 
and suburban schools 
outstripped urban 
schools on this charac- 
teristic. In this case 
however, male students 
were more likely to 
have teachers who had 
received professional 
development than were 
female students. There 
were no significant 
differences based 
upon ethnicity, school 
governance, or 
economic status. 

These results suggest 
that disadvantaged 
groups, to some extent, 
receive less exposure to 
teachers well prepared 
in technology use than 
do other groups of 



Figure 7: Percentage of Fourth-Graders Whose Teachers Report 
Any Professional Development in Technology Use in the Previous 
Five Years 
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Figure 8: Percentage of Eighth-Graders Whose Teachers Report 


Any Professional Development in Technology Use in the Previous 


Five Years 
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students. Suburban and 
affluent fourth-graders 
are more likely to have 
technologically prepared 
teachers than urban and 
poor fourth-graders; 
suburban eighth-graders 
are more likely to have 
technologically prepared 
teachers than urban 
eighth-graders, although 
there are no differences 
based on economic 
status for this grade. 

Types of Computer Use 

The types of use to 
which computers are put 
varies greatly between 
fourth and eighth grade. 
Among fourth-graders, 
54.5 percent have teach- 
ers reporting learning 
games as the primary 
use; 35.9 percent report 
drill and practice; 7.5 
percent report simula- 
tions and applications; 
and 2.1 percent report 
introducing new topics. 
Thus the activity tradi- 
tionally thought of as 
teaching higher-order 
skills — applying con- 
cepts or developing 
simulations to illustrate 
them — is rarely used. 
Given the computational 
nature of fourth-grade 
mathematics, however, 
it may be that whatever 
higher-order thinking 
is occurring is being 
conveyed in learning 
games. Among 
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eighth-graders, drill and 
practice is the most 
common activity, with 
teachers for 34.3 percent 
of students reporting it as 
the primary use. Playing 
learning games is the 
second most common 
use, followed closely by 
applications (29.2 percent 
and 27.2 percent of 
students respectively). 
Teachers report introduc- 
ing new topics as the 
primary use just 9.2 
percent of the time. Thus 
it seems that for fourth 
and eighth grades, 
computers are often used 
for lower-order activities 
(i.e. drill and practice), 
but they are also fre- 
quently used for higher- 
order activities (learning 
games for fourth-graders 
and applications/simula- 
tions for eighth-graders). 

There are, however, 
substantial variations in 
patterns of computer use 
between subgroups. 
Among fourth-graders 
(Figure 9), there are 
differences based upon 
region, school gover- 
nance, and community 
status. Southeastern 
students are less likely 
to play learning games 
and more likely to use 
computers for drill and 
practice. And students in 
public and suburban 
schools are less likely to 
use computers for such 
games, although there 



Fogmire 9: Percentage off Fourth-Graders Whose Teachers Report 
Learning Games and Drill as Primary Computer Uses 
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are no differences for 
drill and practice. There 
are no significant 
differences based 
upon ethnicity, gender, 
or economic status. 

Among eighth- 
graders, there are 
differences in use for 
all categories except 
gender (Figure 10). 
Blacks are less likely 
to use computers for 
applications and more 
likely to use them for 
drill and practice. Use 
varies with region as 
well. The Southeast is 
the most likely to drill 
and the least likely to 
do applications; the 
Midwest is the most 
likely to do applications 
and the least likely to 
drill. Also, suburban and 
non-poor students are 
more likely to use 
computers for applica- 
tions; public school 
students are more likely 
to use computers for 
drill and practice. 

It appears, then, 
that there is little differ- 
ence in computer use 
between advantaged 
and disadvantaged 
groups in fourth grade, 
but great differences in 
eighth grade. In eighth 
grade, minority, poor, 
and urban students are 
more likely to find 
themselves learning 
lower-order skills than 
their White, non-poor, 




Figure 10: Percentage of Eighth-Graders Whose Teachers Report 
Simulations/Applications and Drill/Practices as Primary Computer 
Uses 
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and suburban counter- 
parts; disadvantaged 
students are also 
less likely to find them- 
selves learning higher- 
order skills. 

In summary, some 
indicators of the organi- 
zation of computer use 
in schools raised equity 
issues while others did 
not. Disadvantaged 
groups do seem to be 
less likely to have 
teachers well-prepared 
to use computers, and 
disadvantaged eighth- 
graders seem to be less 
likely to be exposed to 
higher order learning 
through computers. 
Disadvantaged students 
are also less likely to 
come from families that 
own computers. How- 
ever, there seems to be 
no disparity in access to 
computers at school, or 
in their frequency of use 
at home for those 
families that own one. 

It seems that policies 
to promote computer 
access in school have 
succeeded in eliminating 
inequities of this sort; 
yet inequities in teacher 
preparedness and what 
is taught using comput- 
ers remain. 

Identifying some 
disparities does not 
suggest whether these 
disparities matter. The 
importance of the 



disparity for a given 
indicator depends upon 
the overall significance 
of that indicator to 
academic achievement 
and other educational 
outcomes. To determine 
this, it is necessary to 
measure the relationships 
between each indicator 
and these outcomes. 
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Chapter Three: 
Modeling 
Technology’s 
Effectiveness 



As discussed in Chapter 
One, the effectiveness of 
educational technology 
has traditionally been 
measured through 
tabulating the results of 
evaluations of exemplary 
technology programs. 
While such evaluations 
may be useful in provid- 
ing examples of how 
technology can be suc- 
cessfully implemented, 
they suffer from certain 
shortcomings that make it 
difficult to conclude that 
educational technology is 
(or is not) related to 
various educational 
outcomes: The studies do 
not distinguish between 
types of technology use 
or program components; 
they present a picture of 
a particular classroom, 
school, or school district 
rather than a nationwide 
or even statewide pic- 
ture; they often use poor 
measures of educational 
outcomes, particularly of 
academic achievement; 
and they often disregard 
differences in the student 
bodies between technol- 
ogy-rich and technology- 
poor environments. 

How the Study Was 
Conducted 

The current study 
took a different approach 
to the measurement of 
technology’s effective- 
ness. It conducted 
statistical analyses of the 



1996 National Assess- 
ment of Educational 
Progress (NAEP) in 
mathematics for the 
two samples of students 
presented in Chapter 
Two: 6,227 fourth- 
graders and 7,146 eighth- 
graders. Using the 
technique of structural 
equation modeling, the 
study tested a model of 
how various technology 
characteristics might be 
related to various educa- 
tional outcomes. For 
instance, the model 
might expect that profes- 
sional development on 
technology leads teach- 
ers to use computers 
to teach higher-order 
thinking skills, which, 
in turn, improves test 
scores. In addition to such 
a flow chart, the model 
takes into account how 
other nontechnological 
characteristics of schools 
and students might affect 
the educational outcomes: 
The appearance of higher 
test scores in students 
who use technology more 
frequently may be due to 
the technology, or it may 
be due to the fact that 
such students come from 
more affluent families, 
and so are better academi- 
cally prepared in the first 
place. Apparent higher 
achievement levels of 
students with teachers 
who are computer- 
proficient may be due to 
this proficiency, or it 




may be due to these 
same teachers having 
more teaching experi- 
ence and knowledge of 
their subject matter. The 
model is then tested 
against the data, in this 
case from NAEP, to 
discover which relation- 
ships are significant, 
which relationships are 
not, and how strong 
the significant relation- 
ships are. 

For this study, four 
indicators of the organi- 
zation of technology use 
are related to two educa- 
tional outcomes and to 
one another. The four 
aspects of technology are: 

• frequency of school 
computer use for 
mathematical tasks; 

• access to home com- 
puters/frequency of 
home computer use; 

• professional develop- 
ment of mathematics 
teachers in technology 
use; 

• higher-order and 
lower-order uses of 
computers by math- 
ematics teachers and 
their students 

Frequency of school 
computer use is based 
on student reports and 
ranges from “never or 
hardly ever” to “almost 
every day.” Frequency of 
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home computer use is 
the same as school 
computer use, except 
that students have the 
option of reporting 
“there is no computer at 
home,” which is consid- 
ered here the lowest 
possible level of use. 
Professional develop- 
ment is based upon 
asking teachers whether 
they have received such 
development in technol- 
ogy, particularly com- 
puters, in the previous 
five years. And higher- 
order and lower-order 
uses are measured for 
eighth-graders as “simu- 
lations and applications” 
for higher-order skills 
and “drill and practice” 
for lower-order skills; 
for fourth-graders, 
higher-order thinking 
is measured from 
“playing mathematical/ 
learning games.” 

Two outcomes are 
considered: academic 
achievement in math- 
ematics and the social 
environment of the 
school. Academic 
achievement is mea- 
sured by scores on the 
core mathematics 
assessment. The social 
environment is mea- 
sured by bringing 
together various indica- 
tors of how teachers 
and students feel 



about going to school. 
These are: 

° student tardiness; 

° student absenteeism; 

° teacher absenteeism; 

° teacher morale; 
o student regard for 
school property 

All of these indicators 
come from reports by 
principals. 

In addition, three 
characteristics of stu- 
dents and schools are 
taken into account in the 
model. These are: 

o student socioeco- 
nomic status (SES); 
o class size; 

° teacher characteristics 

This means that all 
relationships between 
technology and educa- 
tional outcomes reported 
here represent the value 
added by technology for 
comparable groups of 
students with compa- 
rable teachers in compa- 
rably sized classes. 

For both fourth- and 
eighth-graders, the 
relative affluence of a 
student and the student’s 
family is measured from 
a series of indicators, 
including parent educa- 
tion levels and posses- 
sions in the home. 9 Also 
for both fourth- and 



eighth-graders, class size 
is measured from a 
teacher report of how 
many students are in 
their class. For fourth- 
graders, the teacher 
characteristic measured 
is teacher education level 
(less than B.A., a B.A., 
an M.A., a Ph.D.). For 
eighth-graders, teacher 
education levels are 
combined with years of 
experience and whether 
or not the teacher had 
majored or minored in 
mathematics or math- 
ematics education. 

This approach 
should address the 
shortcomings of earlier 
research mentioned 
above. It distinguishes 
between different 
aspects of the organiza- 
tion of technology use; 
it is national in scope, 
analyzing national 
samples of students; it 
uses robust and well- 
validated measures of 
academic achievement, 
as well as presenting 
some noncognitive 
outcomes; and it takes 
into account differences 
in student bodies, as 
well as school character- 
istics. (For more infor- 
mation on how the 
study was conducted, 
see Appendix.) 



Model Results: Fourth 
Grade 

As seen in Figures 
11 and 12, many of the 
technology characteris- 
tics are related to one 
another. School com- 
puter use is positively 
related both to home 
computer use and 
teacher professional 
development. This 
means that students 
who use computers in 
school with great fre- 
quency are more likely 
to use them at home 
and more likely to have 
teachers who have 
received professional 
development in technol- 
ogy. Professional devel- 
opment in technology 
appears to be related to 
students’ playing learn- 
ing games; this means 
that teachers who have 
had professional devel- 
opment on computers 
are more likely to use 
them in that way. 

These technology 
characteristics appear to 
have different relation- 
ships to the two out- 
comes. In both cases, the 
relationship to learning 
games seems positive, 
while the relationships to 
frequency of school and 
home computer use are 
negative. Professional 
development is also 



9 Some questions have been raised regarding the validity of student reports of these measures (Berends & Koretz, 1995). 
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Figure 11: Influences on Academic Achievement — Fourth-Graders 




Relationships take into account student socioeconomic status, teacher characteristics, and average class size. 



Figure 12: Influences on Social Environment — Fourth-Graders 




Relationships take into account student socioeconomic status, teacher characteristics, and average class size. 
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related to both outcomes, 
but indirectly; by virtue 
of increasing the use 
of learning games, it 
increases academic 
achievement and improves 
the social environment. 

Model Results: Eighth 
Grade 

For eighth-graders, 
the types of technology 
are again related to one 
another. School use is 
positively related to home 
use, indicating that 
students who use com- 
puters frequently in 
school are also more 
likely to use them 



frequently at home. 
Professional develop- 
ment is positively 
related to higher-order 
thinking, suggesting 
that teachers who are 
knowledgeable in the 
use of computers are 
more likely to use 
them for higher-order 
purposes. 

Again, these tech- 
nology characteristics 
appear to have differ- 
ent relationships to 
academic achievement 
and social environment 
(Figures 13 and 1 4). 
School computer use 
is negatively related 
to mathematics 



achievement although, 
in this case, home 
computer use is posi- 
tively related to math- 
ematics achievement. 
Thus, students who use 
computers frequently at 
home demonstrate 
higher levels of aca- 
demic achievement, 
whereas those who use 
computers frequently at 
school evince lower 
levels. Yet since school 
computer use is associ- 
ated with home com- 
puter use, school com- 
puter use does not have 
wholly negative conse- 
quences for academic 
achievement; students 



who use computers at 
school frequently are 
also more likely to use 
them at home fre- 
quently, and the latter 
use is positively related 
to academic achieve- 
ment. The net effect of 
school computer use is 
negative, but it is miti- 
gated by this indirect 
relationship. Profes- 
sional development and 
higher-order thinking 
are both positively 
related to academic 
achievement: Students 
with teachers who have 
had such professional 
development show 
higher levels of 



Figure 13: Influences on Academic Achievement — Eighth-Graders 




Relationships take into account student socioeconomic status, teacher characteristics, and average class size. 
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achievement, as do 
those who are taught 
higher-order skills with 
computers. Professional 
development is also 
associated with aca- 
demic achievement 
indirectly; it increases 
the likelihood of teach- 
ers using computers to 
convey higher-order 
skills, thereby increas- 
ing academic achieve- 
ment in that way. 

Finally, using computers 
for drill and practice, 
the lower-order skills, is 
negatively related to 
academic achievement. 
For social environment, 
the pattern is the same, 



except that school 
computer use is not 
directly related to it; it 
only has the indirect 
benefit of increasing 
home use, which is 
conducive to the social 
environment in school. 

The Size of the 
Relationships 

While the flow 
charts say a great deal 
about the differential 
effectiveness of the 
technology indicators, 
they do not quantify 
that effectiveness. Such 
quantification can be 
accomplished in two 



ways. First, the size of 
the relationships can be 
measured in terms of 
estimated grade levels. 
This is accomplished by 
taking the differences in 
NAEP scores between 
the characteristics and 
dividing by 12.25, the 
typical difference in 
mathematics achievement 
between grade levels. Yet 
this says little about the 
size of the relationships 
relative to other charac- 
teristics of schools and 
students, such as varia- 
tions in class size. Fortu- 
nately, the technology 
characteristics can be 
compared to other 



characteristics in the 
model by standardizing 
them, meaning that they 
are all placed on the 
same scale; the numbers 
that result are meaning- 
less in absolute terms, 
but do show the strength 
of each relationship 
relative to the others. In 
both cases, it should be 
emphasized, these 
numbers represent the 
value added by technol- 
ogy above and beyond 
student, teacher, and 
classroom characteristics. 

The grade-level 
estimates suggest sub- 
stantial positive benefits 
of technology for 



Figure 14: Influences on Social Environment — Eighth-Graders 




Relationships take into account student socioeconomic status, teacher characteristics, and average class size. 
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eighth-graders, but mixed 
results for fourth-graders 
(Table 1). In fourth grade, 
students using computers to 
play learning games are .15 
of a grade level ahead of 
their counterparts; students 
with teachers who have 
had professional develop- 
ment on computers are .09 
of a grade level ahead. 
Assuming a 36-week school 
year, these gains amount to 
three and five weeks 
respectively. In eighth 
grade, students using 
computers for higher-order 
thinking skills show gains 
of .42 of a grade level, and 
those with a teacher who 
has received professional 
development on computers 
show gains of .35 of a 
grade level. These would 
amount to much more 
substantial gains of 15 and 
13 weeks respectively. 

The size of the technol- 
ogy relationship relative to 
other characteristics also 
emphasizes differences 
between fourth- and eighth- 
graders (Table 2). 10 SES has 
the largest relationship to 
achievement, at .58 for 
fourth grade and .40 for 
eighth grade. The sums of 
the class size and teacher 
characteristic relationships 
are much smaller, totaling 
.08 and .13 for fourth- and 
eighth-graders respectively. 



Table 1- Size of Relationships: Grade Levels 



Variable 


Fourth Grade 


Eighth Grade 


Professional 

Development 


.09 


.35 


Primary Use — 
Applications 


n/a 


.42 


Primary Use — 
Games 


.15 


n/a 


Primary Use — 
Drill 


n/a 


-.59 


School Computer 
Use 


-.20 


-.11 


Home Computer 
Use 


-.26 


.14 



Table 2- Size of Relationships: Relative Sizes 



Variable 


Fourth Grade 


Eighth Grade 


Socioeconomic 

Status 


.58 


.40 


Class Size 


.07 


.06 


Teacher 

Characteristics 


.01 


.07 


Professional 

Develoment 


.01 


.04 


Primary Use — 
Applications 


n/a 


.04 


Primary Use — 
Games 


.03 


n/a 


Primary Use — 
Drill 


n/a 


-.06 


School Computer 
Use 


-.09 


-.05 


Home Computer 
Use 


-.14 


.07 



10 The positive relationship between class size and mathematics achievement indicates that smaller classes are associated with higher levels of 
achievement. 
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The benefits of technol- 
ogy are smaller still, at 
.04 for professional 
development and learn- 
ing games in fourth 
grade, and .08 for profes- 
sional development and 
higher-order thinking in 
eighth grade. Yet while 
the impact of technology 
is smaller in both cases, 
it plays a larger role in 
eighth grade than in 
fourth grade; in eighth 
grade it is more than half 
as great as class size and 
teacher characteristics 
and a fifth as large as 
SES; for fourth grade it is 
one-half the size of class 
size and teacher charac- 
teristics, but one-four- 
teenth as large as SES. 

There were also 
differences in relation- 
ship sizes for those types 
of technology that are 
negatively associated 
with academic achieve- 
ment. For fourth-graders, 
these negative relation- 
ships are larger than 
the positive; while the 
positive total .04, the 
negative total .23. On 
the other hand, for the 
eighth-graders the 
positive relationships 
outweigh the negative; if 
using home computers is 



included, the positive 
relationships total .15 
while the negative 
relationships total .11. 

These results, then, 
paint a more complete 
picture of technology’s 
effectiveness. In general, 
the indicators of profes- 
sional development and 
higher-order uses of 
computers seem posi- 
tively related to academic 
achievement in math- 
ematics and other out- 
comes, while frequency 
of use is unrelated or 
even negatively related 
to them. Further, while 
the size of the positive 
relationships are substan- 
tial for eighth-graders, 
they are small for fourth- 
graders, so small that 
they are overshadowed 
by various negative 
technology relationships. 
The following chapter 
suggests what can be 
made of these findings 
from a policy perspective 
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Interpreting the 
Results 

The results from 
this study suggest that, 
as technology advo- 
cates have asserted, 
technology does matter 
to academic achieve- 
ment, with the impor- 
tant caveat that whether 
it matters depends 
upon how it is used. 
The levels of use of 
computers seems not to 
matter, and extremely 
high levels of use may 
even be counterproduc- 
tive. Possibly at such 
high levels students are 
using computers in 
unproductive ways, 
such as playing non- 
educational games. But 
when computers are 
used to perform certain 
tasks, namely applying 
higher order concepts, 
and when teachers are 
proficient enough in 
computer use to direct 
students toward pro- 
ductive uses more 
generally, computers do 
seem to be associated 
with significant gains in 
mathematics achieve- 
ment, as well as an 
improved social envi- 
ronment in the school. 

The findings regard- 
ing home computers 
reinforce this point. 
Students using home 
computers frequently 
had higher levels of 



achievement in eighth 
grade, but lower levels 
of achievement in fourth 
grade. Presumably this 
was so because the 
computer was put to 
different uses in the two 
grades. Perhaps eighth- 
graders have more 
understanding of how to 
use the Internet than 
fourth-graders, and so 
can use it to provide 
supplementary informa- 
tion on academic sub- 
jects; perhaps eighth- 
graders are more likely 
to use computers as 
tools for doing home- 
work, including using 
word processors for 
writing papers and 
spreadsheets for doing 
calculations and tabulat- 
ing data. Without having 
actual data on how 
home computers are 
used by students, it is 
only possible to specu- 
late, but perhaps if 
home computers are 
academically productive 
for eighth-graders 
and not for fourth- 
graders they are being 
used in substantially 
different ways. 

The descriptive 
information in Chapter 
Two, when combined 
with the results of the 
analyses in Chapter 
Three, suggests the 
importance of various 
inequities in technology 
between groups of 
students. Disadvantaged 




groups seem to lag 
behind in access to 
those aspects of 
technology that do 
affect educational 
outcomes, but not in 
access to those aspects 
of technology that do 
not affect educational 
outcomes. While minor- 
ity, poor, and urban 
students are no less 
likely to use computers 
at school frequently, 
frequency of use is not 
associated with gains in 
achievement or social 
environment. Yet minor- 
ity, poor, and urban 
students are less likely 
to receive exposure to 
computers for higher- 
order learning, and poor 
and urban students are 
less likely to have 
teachers who have 
received professional 
development on technol- 
ogy use. Thus, where 
technology matters, there 
are significant inequities; 
only where technology 
does not matter have 
these inequities been 
successfully erased. 

Knowing More 

While this study 
does address problems 
in the prior literature, it 
has its own method- 
ological limitations. 

First, the data used were 
collected at a single 
point in time; technol- 
ogy characteristics occur 



at the same time as the 
educational outcomes. 
There are no prior 
measures of mathematics 
achievement, making it 
difficult to rule out the 
possibility that positive 
educational outcomes 
are conducive to certain 
aspects of technology 
use rather than the other 
way around. The study 
did address this problem 
in part by taking into 
account student back- 
ground, a proxy for 
prior achievement. 

Also, an alternate 
model for eighth-graders 
was developed that took 
into account the level of 
difficulty of the eighth- 
grade mathematics class 
(algebra, pre-algebra, or 
regular eighth-grade 
mathematics). Since the 
class that students take 
depends to some degree 
on prior achievement, 
this measure may also 
serve as a proxy for it. 
The results from the 
alternate model, how- 
ever, did not differ much 
from those of the original 
model. All technology 
measures remained 
significantly related to 
academic achievement 
and the social environ- 
ment of the school, and 
the sizes and directions 
of the relationships also 
remained the same. The 
one exception was 
professional develop- 
ment, which, while still 
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significantly and posi- 
tively associated with the 
educational outcomes, 
was associated with them 
to a somewhat lesser 
extent, with a relation- 
ship size of .21 grade 
levels (as opposed to 
.35 grade levels). The 
relative lack of change 
between the two models 
suggests that a study that 
took prior achievement 
into account would be 
unlikely to produce very 
different results. None- 
theless, a study that 
follows students over 
time, measuring their 
academic achievement 
both before and after 
their exposure to various 
uses of technology 
would provide a stronger 
test of the findings from 
this study. 

Second, while the 
study does take into 
account teacher charac- 
teristics, such as educa- 
tion levels, it does not 
take into account more 
detailed measures of 
teacher practices. The 
study found that when 
computers were used for 
higher-order thinking 
skills, students per- 
formed better; yet this 
does not exclude the 
possibility that students 
do better when teachers 
tend to teach higher- 
order thinking skills, 
regardless of the 
medium. Fortunately, 
this invitation does not 




undermine the findings 
of this study. Regardless 
of whether other media 
may also be successful at 
conveying higher-order 
thinking, this study does 
demonstrate that com- 
puters can do so; they 
are, at a minimum, one 
of a limited number of 
tools that can contribute 
to this goal. 

A few other caveats 
are worth noting. The 
study only analyzes 
technology’s effective- 
ness in one subject area, 
mathematics. Most 
measures of the uses of 
technology as well as the 
measure of academic 
achievement are specific 
to that subject. Comput- 
ers may be less effective 
in other subject areas, 
such as history and 
English. Also, the study 
is limited to only a few 
indicators of the organi- 
zation of technology use; 
other aspects of technol- 
ogy may be more or less 
effective. For instance, 
the study does not 
distinguish between the 
effectiveness of different 
types of software. 

Finally, the study 
does not incorporate 
information about state 
technology policies, 
which may have a 
bearing on technology’s 
effectiveness. These 
three caveats all suggest 
that it is important for 
states to collect more 



data to evaluate their 
technology investments; 
they need to know 
which policies and types 
of software are effective, 
and for which subject 
areas computers should 
be used. 

Implications of Findings 

These methodologi- 
cal limitations aside, the 
findings have significant 
implications for technol- 
ogy policy and practice. 

First, the study 
suggests that federal 
and state policymakers 
should redouble their 
efforts to ensure that 
teachers are properly 
trained to use computers. 
The descriptive informa- 
tion supplied in Chapter 
Two shows that the 
vast majority of teachers 
already receive some 
professional development 
in technology, suggesting 
that there is little room 
for policymakers to 
further increase such 
professional develop- 
ment. Yet there are large 
disparities in the amount 
of professional develop- 
ment teachers receive. 
Federal and state policy- 
makers should target 
teacher training efforts at 
high-poverty urban and 
rural schools, which have 
much lower percentages 
of teachers receiving such 
training. Also, profes- 
sional development in 



technology, as reported 
in NAEP, could refer to 
anything from a week- 
end seminar to a semes- 
ter-long course. Many 
of the teachers who do 
receive professional 
development in technol- 
ogy may benefit from 
more intensive instruc- 
tion. Federal and state 
policymakers should 
make sure that the 
quality of the teacher 
training offered is high 
and intensive, since 
this training is such an 
important component 
of making technology 
use successful. 

Second, the study 
suggests that teachers 
should focus on using 
computers to apply 
higher-order skills 
learned elsewhere in 
class. Computers should 
be a component of 
a seamless web of 
instruction that includes 
nontechnological com- 
ponents. For example, 
teachers might introduce 
new topics and convey 
basic information to 
their students through 
general class discussion 
and lecture, then assign 
projects and problems 
that computers as well 
as other media (books, 
field trips, etc.) can be 
used to address. While 
implementing this vision 
depends primarily upon 
the individual prefer- 
ences and capabilities 
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of teachers, federal and 
state policymakers can 
encourage it to some 
degree through sup- 
porting professional 
development of teach- 
ers that emphasizes 
this teaching method. 
Further, federal and 
state policymakers can 
encourage school 
districts to order soft- 
ware that draws on 
these skills, and assist 
school districts in 
connecting to the 
Internet, which can be 
used to obtain outside 
information for projects 
and problem solving. 

As with teacher devel- 
opment, extra efforts 
should be made to 
support these changes 
in high-poverty urban 
and rural school dis- 
tricts, which are at 
present most likely to 
be using computers for 
drill and practice rather 
than applications and 
simulations. This is not 
to say that the basics 
should not be taught 
where appropriate; the 
findings merely suggest 
that computers are not 
well-adapted to teach- 
ing them. 

Third, the primary 
focus of all technology 
initiatives should be 
on middle schools 
rather than elementary 
schools. The effects of 
technology appear to 
be much smaller in the 
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fourth than the eighth 
grade, and so may not 
be cost-effective. Fur- 
ther, the sequence of the 
typical mathematics 
curriculum suggests that 
computers are more 
crucial for middle school 
students than elementary 
school students. Most 
higher-order concepts 
are not introduced 
before middle school, 
with elementary school 
students focusing on 
computational skills. 

To the extent that the 
primary benefit of 
computers lies in apply- 
ing higher order skills, 
there may not be much 
opportunity to benefit 
from using computers 
before middle school. 

In conclusion, 
technology advocates 
were correct in asserting 
that technology can be 
beneficial to student 
learning. Used properly, 
technology can lead 
to gains in academic 
achievement and posi- 
tively influence the 
social environment of 
the school, reducing 
teacher and student 
absenteeism and increas- 
ing morale. Yet it is 
important that the scope 
of technology in schools 
be limited to those areas 
where it provides ben- 
efits, and reduced in 
areas where it does 
not. Thus the notion 
of technology as a 




substitute for conven- 
tional forms of instruc- 
tion, often hoped for by 
technology advocates 
and dreaded by technol- 
ogy critics, may over- 
state the case for tech- 
nology use. It may also 
be counterproductive, 
not only introducing 
computers into areas 
where other teaching 
techniques are more 
conducive to positive 
educational outcomes, 
but also raising worries 
among teachers that 
they are being replaced 
by machines. By clearly 
delineating areas in 
which computers can 
be helpful to teachers 
and areas in which 
they cannot, it will be 
possible to increase the 
acceptance of comput- 
ers. Alongside chalk 
and blackboards, com- 
puters will be tools 
teachers feel they 
cannot live without. 
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Appendix: 
How The 
Study Was 
Conducted 



Case and Variable Selection 

Data were drawn 
from the 1996 National 
Assessment of Educational 
Progress in mathematics 
for fourth- and eighth- 
graders. For each grade, 
student and teacher data 
were linked to school data 
using the school identifier, 
SCRPSU. Only those cases 
which had plausible values 
for the core assessments, 
MRPCM1, MRPCM2, 
MRPCM3, MRPCM4, 
and MRPCM5, were used, 
resulting in 6,227 fourth- 
graders and 7,146 
eighth-graders. 

To measure socio- 
economic status the follow- 
ing six variables were 
selected: B003501M 
(mother’s education), 
B003601M (father’s educa- 
tion), B000901M (newspa- 
pers), B000903M (encyclo- 
pedias), B000904M (books), 
and B000905M (magazines). 

To measure teacher 
characteristics, the follow- 
ing six variables were 
selected: T040301 (years 
taught), T056301 (highest 
degree), T040703 (math- 
ematics major), T040704 
(mathematics education 
major), T040803 (math- 
ematics minor), and 
T040804 (mathematics 
education minor). Class 
size was measured from 
T044000. Academic 
achievement was 
measured from the five 



plausible value variables 
used for case selection. 

To measure social 
environment, the follow- 
ing five variables were 
selected: C032402 (stu- 
dent absenteeism), 
C032401 (student tardi- 
ness), C032406 (teacher 
absenteeism), C032502 
(teacher morale), and 
C032506 (regard for 
school property). 

With regard to the 
technology variables, 
frequency of school 
computer use was taken 
from M812710B; fre- 
quency of home com- 
puter use was taken 
from B009301M; teacher 
development was taken 
from T056702; and 
primary use was taken 
from T057601. All vari- 
ables were recoded to 
eliminate missing values. 

Whether teachers 
had received education 
in mathematics, one of 
the teacher characteris- 
tics, was calculated by 
treating a “yes” response 
to any of T040703, 
T040704, T040803, or 
T040804 as having 
received such education. 

The two frequency- 
of-computer-use vari- 
ables were recoded so 
that a high value de- 
noted high frequency. 
The class-size variable 
was recoded so that a 
small class had a high 
value, and the social 



environment measures 
were recoded so that 
a high value indicated 
a positive social 
environment. 

Descriptive Analyses 

All cases were 
weighted by the variable 
ORIGWT divided by the 
mean weight for each 
sample. The overall 
number of cases was 
also divided by a design 
effect of 1.75. Compari- 
sons of means were then 
conducted using the 
Scheffe test. 

Multivariate Analyses 

Models of the rela- 
tionship between tech- 
nology characteristics 
and educational 
outcomes were devel- 
oped based upon the 
literature. The models 
were tested for fourth- 
and eighth-graders 
separately using the 
technique of Structural 
Equation Modeling as 
implemented by two 
software packages: 
STREAMS 2.0 and AMOS 
3.6; STREAMS is a pre- 
and postprocessor for 
AMOS. Unidimensional 
measurement models 
were developed for 
academic achievement, 
social environment, SES, 
and teacher characteris- 
tics based upon their 
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corresponding variables 
in the database. For 
the remaining variables, 
the constaicts were 
defined as the database 
variables with a factor 
loading of 1 and no 
measurement error. 

The constructs were 
related to one another as 
follows: for fourth- 
graders, the technology 
indicators, class size, 
teacher characteristics, 
and SES were all related 
to academic achievement 
and school social envi- 
ronment. Professional 
development was related 
to use of learning games. 
Social environment was 
related to academic 
achievement. SES was 
related to frequency of 
school computer use and 
both of those were 
related to home com- 
puter use. Frequency of 
school computer use was 
related to professional 
development. For eighth- 
graders the model was 
the same, except that 
frequency of school 
computer use was not 
related to professional 
development and profes- 
sional development was 
related to simulation/ 
applications and drill and 
practice rather than 
learning games. Models 
were deemed acceptable 
with goodness of fit 
indices better than .9 
and a root mean squared 



error of approximation 
better than .05. Coeffi- 
cients were deemed 
significant with stan- 
dard errors inflated by 
the square root of the 
design effect (1.75) at 
the .05 level. Sensitivity 
analyses were con- 
ducted in which stan- 
dard errors were further 
inflated by the measure- 
ment variability of the 
plausible values, and 
the measurement error 
in the measurement 
model was correspond- 
ingly eliminated; this 
procedure produced 
similar results. For 
STREAMS 2.0, see 
Gustafsson and Stahl 
(1997). For AMOS 3.6 
see Arbuckle (1997). For 
discussions of weight- 
ing, design effects, and 
measurement variability 
using NAEP, see 
Johnson (1989), and 
Johnson, Mislevy, & 
Thomas (1994). 
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